大量数据集和高容量模型推动了计算机视觉和自然语言理解方面的许多最新进步。这项工作提出了一个平台，可以在体现的AI中实现类似的成功案例。我们提出了Procthor，这是一个程序生成体现的AI环境的框架。 Procthor使我们能够采样多种，交互式，可自定义和性能的虚拟环境的任意大型数据集，以训练和评估在导航，互动和操纵任务中的体现代理。我们通过10,000个生成的房屋和简单的神经模型的样本来证明procthor的能力和潜力。仅在Procthor上仅使用RGB图像训练的模型，没有明确的映射，并且没有人类任务监督在6个体现的AI基准中产生最先进的结果，用于导航，重排和手臂操纵，包括目前正在运行的Habitat 2022，AI2-- Thor重新安排2022，以及机器人挑战。我们还通过对procthor进行预训练，在下游基准测试上没有进行微调，通常会击败以前的最先进的系统，从而访问下游训练数据。

语言模型既展示了定量的改进，又展示了新的定性功能，随着规模的增加。尽管它们具有潜在的变革性影响，但这些新能力的特征却很差。为了为未来的研究提供信息，为破坏性的新模型能力做准备，并改善社会有害的效果，至关重要的是，我们必须了解目前和近乎未来的能力和语言模型的局限性。为了应对这一挑战，我们介绍了超越模仿游戏基准（Big Bench）。 Big Bench目前由204个任务组成，由132家机构的442位作者贡献。任务主题是多样的，从语言学，儿童发展，数学，常识性推理，生物学，物理学，社会偏见，软件开发等等。 Big-Bench专注于被认为超出当前语言模型的功能的任务。我们评估了OpenAI的GPT型号，Google内部密集变压器体系结构和大型基础上的开关稀疏变压器的行为，跨越了数百万到数十亿个参数。此外，一个人类专家评估者团队执行了所有任务，以提供强大的基准。研究结果包括：模型性能和校准都随规模改善，但绝对的术语（以及与评估者的性能相比）；在模型类中的性能非常相似，尽管带有稀疏性。逐渐和预测的任务通常涉及大量知识或记忆成分，而在临界规模上表现出“突破性”行为的任务通常涉及多个步骤或组成部分或脆性指标；社交偏见通常会随着含糊不清的环境而随着规模而增加，但这可以通过提示来改善。

知识跟踪是跟踪给定学习领域的学生不同技能的掌握程度的过程。它是建立自适应学习系统的关键组件之一，并已被调查几十年。与其他领域的深度神经网络的成功平行，我们看到研究人员在学习科学界采取类似的方法。但是，大多数现有的深度学习知识追踪模型：（1）仅使用正确/不正确的响应（忽略来自其他方式的有用信息）或（2）通过试验和错误通过域专业知识设计其网络架构。在本文中，我们提出了一种基于模型的基于模型的优化方法，该优化方法结合了一个框架内的多峰融合和神经结构。当涉及一个模态时，常用的神经结构搜索技术可以被认为是我们所提出的方法的特殊情况。我们进一步建议在曲线（加权AUC）下使用称为时间加权区域的新度量来测量序列模型如何随时间执行。我们在两个公共实时数据集中评估我们的方法，显示发现模型能够实现卓越的性能。与大多数现有的作品不同，我们对McNemar对模型预测的测试进行了测试，结果是统计学意义。

高斯工艺（GP）是一种广泛的工具，用于依次优化黑框功能，其中评估昂贵且潜在的嘈杂。关于GP土匪的最新作品提议超越随机噪声，并设计算法对对抗性攻击的强大。本文从攻击者的角度研究了这个问题，提出了各种对抗性攻击方法，对攻击者的力量和先前信息的假设有所不同。我们的目标是从理论和实践的角度了解对GP土匪的对抗性攻击。我们主要关注对流行的GP-UCB算法的有针对性攻击和基于消除的算法的相关算法，基于对抗性扰动该函数$ f $以产生另一个函数$ \ tilde $ \ tilde {f} $，其Optima在某些目标区域中$ \ \ Mathcal {r} _ {\ rm target} $。根据我们的理论分析，我们设计了白盒攻击（已知$ F $）和黑盒攻击（未知$ F $），前者包括减法攻击和裁剪攻击，以及后者，包括激进的减法攻击。我们证明，对GP土匪的对抗性攻击也可以成功强迫该算法向$ \ Mathcal {R} _ {\ rm Target} $，即使在低攻击预算的情况下，我们也可以测试攻击对各种客观功能的有效性。

我们介绍了互动室（Thor），这是一个视觉AI研究的框架，可在http://ai2thor.allenai.org上找到。AI2-这是由几乎逼真的3D室内场景组成的，在该场景中，AI代理可以在场景中导航并与对象进行交互以执行任务。AI2-这可以在许多不同的领域进行研究，包括但不限于深入强化学习，模仿学习，通过互动，计划，视觉问答答案，无监督的表示学习，对象检测和细分以及认知模型。AI2的目的是促进构建视觉上智能模型，并将研究推向该领域。

Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited several support examples. In this work, we explore a simple yet unified solution for FSIS as well as its incremental variants, and introduce a new framework named Reference Twice (RefT) to fully explore the relationship between support/query features based on a Transformer-like framework. Our key insights are two folds: Firstly, with the aid of support masks, we can generate dynamic class centers more appropriately to re-weight query features. Secondly, we find that support object queries have already encoded key factors after base training. In this way, the query features can be enhanced twice from two aspects, i.e., feature-level and instance-level. In particular, we firstly design a mask-based dynamic weighting module to enhance support features and then propose to link object queries for better calibration via cross-attention. After the above steps, the novel classes can be improved significantly over our strong baseline. Additionally, our new framework can be easily extended to incremental FSIS with minor modification. When benchmarking results on the COCO dataset for FSIS, gFSIS, and iFSIS settings, our method achieves a competitive performance compared to existing approaches across different shots, e.g., we boost nAP by noticeable +8.2/+9.4 over the current state-of-the-art FSIS method for 10/30-shot. We further demonstrate the superiority of our approach on Few Shot Object Detection. Code and model will be available.

Non-linear state-space models, also known as general hidden Markov models, are ubiquitous in statistical machine learning, being the most classical generative models for serial data and sequences in general. The particle-based, rapid incremental smoother PaRIS is a sequential Monte Carlo (SMC) technique allowing for efficient online approximation of expectations of additive functionals under the smoothing distribution in these models. Such expectations appear naturally in several learning contexts, such as likelihood estimation (MLE) and Markov score climbing (MSC). PARIS has linear computational complexity, limited memory requirements and comes with non-asymptotic bounds, convergence results and stability guarantees. Still, being based on self-normalised importance sampling, the PaRIS estimator is biased. Our first contribution is to design a novel additive smoothing algorithm, the Parisian particle Gibbs PPG sampler, which can be viewed as a PaRIS algorithm driven by conditional SMC moves, resulting in bias-reduced estimates of the targeted quantities. We substantiate the PPG algorithm with theoretical results, including new bounds on bias and variance as well as deviation inequalities. Our second contribution is to apply PPG in a learning framework, covering MLE and MSC as special examples. In this context, we establish, under standard assumptions, non-asymptotic bounds highlighting the value of bias reduction and the implicit Rao--Blackwellization of PPG. These are the first non-asymptotic results of this kind in this setting. We illustrate our theoretical results with numerical experiments supporting our claims.

While the capabilities of autonomous systems have been steadily improving in recent years, these systems still struggle to rapidly explore previously unknown environments without the aid of GPS-assisted navigation. The DARPA Subterranean (SubT) Challenge aimed to fast track the development of autonomous exploration systems by evaluating their performance in real-world underground search-and-rescue scenarios. Subterranean environments present a plethora of challenges for robotic systems, such as limited communications, complex topology, visually-degraded sensing, and harsh terrain. The presented solution enables long-term autonomy with minimal human supervision by combining a powerful and independent single-agent autonomy stack, with higher level mission management operating over a flexible mesh network. The autonomy suite deployed on quadruped and wheeled robots was fully independent, freeing the human supervision to loosely supervise the mission and make high-impact strategic decisions. We also discuss lessons learned from fielding our system at the SubT Final Event, relating to vehicle versatility, system adaptability, and re-configurable communications.

We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io